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Abstract 

Objectives:  This  study  estimated  the  validity  of  23  groups  of  codes  in  the 
International  Classification  of  Diseases,  9th  revision,  Clinical  Modification  (ICD- 
9-CM  or  ICD),  for  flagging  adverse  events  (AEs)  related  to  hospital  surgeries. 
Methods:  A  set  of  ICD  codes  selected  as  “flags”  for  in-hospital  AEs  were 
developed  with  the  assistance  of  a  national  expert  panel.  The  codes  were  grouped 
into  66  AE  classes,  23  of  which  were  identified  as  potentially  surgery-related.  The 
predictive  value  positive  (PVP)  of  the  23  classes  was  assessed  by  medical  record 
review  of  941  surgical  discharges.  Results:  The  23  classes  identified  AEs 
associated  with  care  management  during  the  hospital  stay  with  an  average  PVP  of 
45  percent.  They  identified  AEs  specifically  related  to  surgery  with  a  PVP  of  37 
percent.  Conclusions:  This  exploratory  study  identified  a  small  number  of  classes 
and  individual  ICD  codes  that  identified  surgery-related  AEs.  Further  study  of 
selected  codes  is  needed  to  refine  the  classes  and  fully  evaluate  their  use  for 
statewide  surveillance  of  surgery  AEs. 


Introduction 

Identifying  surgical  adverse  events  using  ICD-9-CM  codes 

International  Classification  of  Diseases,  9th  revision,  Clinical  Modification 
(ICD-9-CM  or  ICD)  codes  have  been  evaluated  for  a  variety  of  purposes  beyond 
documenting  health  conditions  and  procedures  to  support  hospital  billing.  Uses 
have  included  characterizing  the  anatomical  location  of  injury,  injury  severity,^ 
disease  severity,4  comorbidity  severity,5  the  likelihood  of  death,6  and  the  presence 
of  medically  caused  injury.7  11  The  resulting  severity  scores,  in  turn,  have  largely 
been  used  to  predict  resource  use  "  and  to  control  for  patient  mix. 

ICD-coded  hospital  discharge  data  have  also  been  used  to  assess  quality  of 
care.  They  are  easily  accessed  through  electronic  databases  and  tend  to  be 
available  in  statewide  repositories.  Research  on  the  use  of  hospital  administrative 
data  for  evaluating  quality  of  care  has  produced  mixed  results,  depending  on  the 
purpose  of  the  researcher.  Efforts  have  been  made  to  identify  complications, 
complications  due  to  medical  care,  complications  due  to  errors  in  medical  care, 
complications  resulting  in  harm,  preventable  complications,  process  errors,  and 
near  misses. 
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Studies  of  the  validity  of  using  ICD  codes  to  identify  adverse  events  (AEs) 
due  to  medical  care  have  frequently  involved  the  Complications  Screening 
Program  (CSP).  ICD  codes  for  identifying  surgical  AEs  have  been  found  to  have 
some  validity,  while  ICD  codes  identifying  AEs  in  medical  patients  have  been 
less  successful.7, 10,  n’ 14  The  diagnosis  and  extemal-cause-of-injury  codes  (E- 
codes)  specifically  designated  for  recording  misadventures  and  complications  of 
surgical  and  medical  care  (996-999,  E870-E879,  E930-E949)  have  also  been 
used  to  study  the  trends  and  patterns  of  AEs.15  Building  on  the  CSP  research,  the 
Agency  for  Healthcare  Research  and  Quality  (AHRQ)  recently  introduced  the 
Patient  Safety  Indicators  (PSIs).16  The  PSIs  are  primarily  based  on  ICD  codes  for 
specific  patient  populations.  They  are  intended  to  flag  patient  safety  problems  and 
point  to  areas  for  investigation  and  intervention.  Zhan  and  Miller  found  they 
could  be  used  to  identify  patients  who  are  more  likely  to  die,  have  longer  lengths 
of  stay,  and  have  higher  charges.9 

In  the  study  reported  here,  ICD  codes  representing  diagnoses,  procedures,  and 
E-codes  were  evaluated  for  their  ability  to  detect  harm  resulting  from  care 
management  of  surgical  patients  in  acute  care  hospitals.  No  attempt  was  made  to 
assess  preventability,  medical  negligence,  or  error.  The  emphasis  was  on 
determining  whether  groups  of  ICD  codes  could  be  used  for  statewide 
surveillance  of  patient  harm  resulting  from  care  during  hospital  stays. 

This  study  is  the  result  of  a  joint  project  between  the  Utah  Department  of 
Health  (UDOH)  and  the  Missouri  Department  of  Health  and  Senior  Services 
(MDHSS)  under  a  cooperative  agreement  with  AHRQ.  The  collaborative  study 
addressed  the  accuracy  and  efficacy  of  using  ICD  codes,  available  in  the 
mandated  discharge  data  collection  systems  in  both  States,  for  surveillance  and 
reporting  on  categories  or  classes  of  adverse  events.  UDOH  evaluated  ICD  codes 
selected  for  their  potential  to  identify  adverse  drug  events  (ADEs).  For  MDHSS, 
the  focus  was  on  AEs  related  to  surgical  hospital  stays. 


Methods 

ICD-9-CM  codes  suggestive  of  surgical  adverse  events 

An  adverse  event  was  considered  to  be  an  undesirable  and  unintended  injury 
resulting  from  medical  care  management  (an  act  of  care  provided  by  the  hospital 
or  the  omission  of  necessary  care,  rather  than  from  a  patient’s  underlying  disease 
process.)  Harm  was  defined  as  death,  prolonged  hospital  stay,  or  temporary  or 
permanent  impairment  of  bodily  function  or  structure  that  requires  intervention 
such  as  a  change  in  monitoring  of  the  patient’s  condition,  a  change  in  therapy,  or 
active  medical  or  surgical  treatment  or  attention  (if  feasible  or  possible).  Both 
harm  and  the  degree  to  which  the  AEs  were  due  to  care  management  were 
operationalized  by  rating  scales  (Appendixes  2  and  3  )  in  the  data  collection  tool. 


Appendixes  cited  in  this  report  are  available  electronically  at  www.dhss.mo.gov/SurgeryAEs. 
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Surgery  AEs  were  defined  as  harm  events  related  to  the  preoperative  care,  the 
surgery,  or  the  postoperative  care. 

To  assist  with  refining  the  ICD-9-CM  classification  of  AEs,  the  Utah/Missouri 
project  team  convened  a  national  expert  panel  comprised  of  physicians,  medical 
record  coders,  pharmacy  directors,  and  health  care  researchers.  An  initial  set  of 
1,192  ICD  codes  were  selected  for  the  study  based  on  the  UDOH  report,  Adverse 
Events  Related  to  Medical  Care,  Utah:  1995-99,  a  review  of  the  literature,  and 
other  research  in  progress  on  the  use  of  ICD  codes  as  flags  of  in-hospital  AEs. 
Following  an  orientation  session  and  consensus  building  on  the  definition  of  AEs 
for  this  study,  each  panelist  reviewed  a  worksheet  including  a  subset  of  the  codes 
with  a  description  of  each  code.  Each  code  was  rated  by  a  minimum  of  three  and  a 
maximum  of  nine  panelists  along  three  scales:  medical  care/causality,  patient 
harm,  and  preventability.  A  total  of  1,003  were  retained  for  the  study.  With 
consultation  from  some  expert  panelists,  the  UDOH  medical  epidemiologists 
grouped  the  final  list  of  codes  into  66  classes.  Codes  selected  as  flags  for  surgery- 
related  AEs  numbered  377  and  made  up  23  adverse  event  classes  (Appendix  l).17 
Additional  details  on  the  Utah/Missouri  ICD-9-CM  classification  of  adverse 
events,  including  the  national  panel  exercise,  will  be  addressed  in  a  separate 
paper. 

Sample  design 

Hospital  discharge  records  for  the  first  three  quarters  of  2001  formed  the 
sampling  frame  for  the  study.  Missouri  hospitals  and  ambulatory  surgery  centers 
have  been  required  by  law  to  submit  records  to  the  MDHSS  since  1993.  Included 
on  the  records  are  patient  identifiers,  demographic  infonnation,  up  to  23 
diagnoses  and  20  procedures,  type  of  admission,  discharge  disposition,  and  E- 
codes.  In  2001  there  were  123  general  acute  care  hospitals  accounting  for  874,836 
inpatient  discharges. 

A  convenience  sample  of  40  hospitals  was  initially  selected  to  provide  data  for 
this  study  and  a  second  study  designed  to  take  advantage  of  the  hospital  license 
renewal  surveys.  Three  of  the  40  hospitals  elected  not  to  participate  in  the  project, 
and  a  fourth  was  dropped  due  to  scheduling  problems  with  the  licensure  survey. 
The  remaining  36  hospitals  comprised  the  sampling  frame  for  both  studies.  They 
tended  to  be  somewhat  larger  and  more  urban  than  Missouri  acute  care  hospitals 
as  a  whole.  Forty-two  percent  had  over  7,000  discharges  (medical  and  surgical), 
compared  to  32  percent  of  all  123  hospitals.  Fifty-six  percent  of  the  36  study 
hospitals  were  in  Metropolitan  Statistical  Areas,  compared  to  49  percent  of 
hospitals  statewide. 

Given  the  large  number  of  codes  selected  for  the  AE  classes,  as  well  as 
constraints  on  the  number  of  medical  records  that  could  be  abstracted  for  this 
study,  the  sample  design  focused  on  representing  the  AE  classes  rather  than 
individual  codes  or  hospitals.  It  was  initially  estimated  that  approximately  1,000 
surgical  discharge  records  would  allow  a  reasonable  trade-off  between  an 
adequate  statistical  analysis  of  the  23  classes  and  a  cost-manageable  number  of 
chart  reviews.  Records  were  sampled  using  the  following  criteria:  First,  only 
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records  with  a  surgical  Diagnostic  Related  Group  (DRG)  were  included.18 
Second,  at  least  one  of  the  flag  codes  had  to  be  present  in  a  secondary  diagnosis 
field  on  the  discharge  record.  (The  principal  diagnosis  was  assumed  to  represent  a 
health  condition  existing  at  the  time  of  admission  and  was  not  searched  for  flag 
codes.)  Finally,  the  record  was  associated  with  a  specific  AE  class  based  on  the 
first  flag  code  encountered  when  searching  through  the  22  secondary  diagnosis 
and  20  procedure  fields  on  the  discharge  record. 

The  sampling  design  was  a  doubly  stratified  random  sample  of  the  23  AE 
classes.  Each  class  was  assigned  to  contribute  a  maximum  of  30,  45,  or  60 
records,  based  on  the  number  of  records  available  for  a  class.  Within  each  class, 
the  hospitals  were  ranked  according  to  the  number  of  records  that  contained  codes 
for  that  class.  The  hospitals  were  then  divided  among  three  roughly  equal 
terciles — the  top  tercile  containing  hospitals  with  the  most  records  for  that  class, 
the  middle  containing  hospitals  with  a  medium  number  of  records,  and  the  bottom 
tercile  containing  hospitals  with  the  fewest  records  for  the  class.  A  hospital  could 
be  assigned  to  the  top  tercile  in  one  class  and  the  middle  or  bottom  tercile  in  a 
different  class.  Each  tercile  was  sampled  proportionately  to  the  percentage  of 
records  it  contained  for  a  class.  Within  these  constraints,  the  records  were  selected 
randomly  and  without  replacement.  Hospitals  having  no  records  with  a  flag  code 
in  a  given  class  were  not  sampled  for  that  class.  This  double  stratification  design 
attempted  to  maximize  the  possibility  that  all  classes  and  all  strata,  but  not  all 
hospitals,  contributed  records. 

Due  to  chance  factors,  four  hospitals  did  not  contribute  records,  leaving  32 
hospitals  and  991  discharges  in  the  final  sample.  Some  attrition  occurred  during 
data  collection  due  to  factors  such  as  unavailable  charts,  inadequate  medical 
record  documentation  for  the  data  abstraction,  and  errors  in  the  specification  of 
the  record  as  an  acute  care  discharge.  At  the  conclusion  of  the  chart  review,  941 
records  representing  150  flag  codes  were  available  for  analysis  of  the  surgery 
AE  classes. 

Identifying  surgical  adverse  events  using 
medical  record  abstraction 

The  data  collection  tool  was  a  computerized  form  adapted  from 
Woloshynowych  et  al.19  The  tool  was  preloaded  with  all  diagnosis  codes,  E- 
codes,  procedure  codes,  and  other  items  taken  from  the  selected  discharge  record. 
Abstractors  recorded  whether  the  identified  injury  or  complication  was  an  AE, 
whether  it  originated  prior  to  or  after  the  current  admission,  and  whether  it  related 
to  surgery.  After  documenting  the  AE,  abstractors  recorded  that  the  AE  was 
associated  with  one  of  the  preloaded  (billed)  ICD  codes  or  with  an  ICD  code  not 
contained  in  the  discharge  record,  or  that  the  AE  was  not  adequately  described  by 
any  existing  ICD  code.  The  form  provided  for  documenting  up  to  three  confirmed 
adverse  events  and  recording  the  relationship,  if  any,  between  the  AEs  and  the 
preloaded  diagnosis  codes,  E-codes,  or  procedure  codes.  To  facilitate  review,  the 
chart  abstraction  tool  contained  the  study  definitions  of  adverse  event  and  hann, 
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and  required  the  reviewer  to  use  scales  to  rate  the  level  of  harm  (Appendix  2)  and 
the  degree  to  which  the  AE  was  due  to  care  management  (Appendix  3). 

The  medical  chart  review  activity  was  conducted  by  Primaris,  the  Medicare 
quality  improvement  organization  for  the  State  of  Missouri.  Nurses  experienced 
in  medical  record  review  were  recruited  to  conduct  the  chart  abstractions  for  the 
study.  Qualifications  of  the  nurse  reviewers  included  3  years  of  experience  with 
clinical  data  abstraction  in  combination  with  either  acute  care  or  clinical  coding 
experience. 

The  nurse  reviewers  were  provided  40  hours  of  specialized  training,  focusing 
on  adverse  event  identification,  study  definitions,  record  abstraction,  ICD  clinical 
coding,  data  collection  tool  completion,  and  data  entry.  A  registered  health 
information  administrator  provided  the  training  on  medical  record  coding  and 
coding  of  adverse  events.  Special  emphasis  was  placed  on  how  and  when  to  use 
E-codes. 

Each  nurse  reviewer  participated  in  a  gold-standard  testing  process  prior  to 
field  assignment  for  the  project  chart  abstractions.  Gold-standard  test  cases 
included  one  case  in  which  no  AE  was  documented  and  four  cases  that  included 
one  or  more  events  that  were  consistent  with  the  study  definition  of  an  AE.  Only 
reviewers  who  passed  the  gold-standard  test  were  retained  for  the  project. 


Analysis 

A  given  AE  could  be  associated  with  more  than  one  ICD  code  on  the  hospital 
discharge  record,  and  one  or  more  of  these  codes  could  belong  to  the  same  class. 
If  a  record  was  sampled  for  a  given  class,  and  the  record  contained  more  than  one 
code  from  that  class,  then  each  code  in  that  class  was  validated  (examined  for  its 
association  with  an  AE).  Each  record  was  used  to  validate  codes  in  only  one  class. 
The  predictive  value  positive  (PVP)  for  a  class  was  calculated  as  the  percentage 
of  codes  in  that  class  that  were  found  to  be  associated  with  an  AE. 

Table  1  shows  the  23  AE  classes  and  the  PVP  of  each  class.  The  PVP:  Post- 
admit  AE  column  displays  the  percentage  of  flag  codes  in  each  class  that  were 
confirmed  by  chart  review  as  referring  to  AEs  that  originated  after  the  patient  was 
admitted  to  the  hospital.  These  patients  were  considered  to  have  incurred  a  patient 
injury  event  (score  of  1  or  more  on  the  Harm  Rating  Scale — Appendix  2)  due  to 
care  management  (score  of  4  or  higher  on  the  Care  Management  Causation  Rating 
Scale — Appendix  3),  following  admission  to  the  hospital.  The  PVP:  Surgery  AE 
column  shows  the  percentage  of  flag  codes  in  each  class  that  met  the  above 
criteria  and  were  also  judged  to  be  related  to  the  surgery,  the  preoperative  care,  or 
the  postoperative  care.  The  Pre-admit  event  column  shows  the  percentages  of  flag 
codes  that  were  not  confirmed  because  the  event  to  which  they  referred  originated 
prior  to  the  current  admission.  The  No  AE  column  shows  the  percentage  of  flag 
codes  that  referred  to  health  conditions  that  failed  to  meet  the  study  criteria  for 
hann  and  care  management  causation.  The  percentages  for  each  adverse  event 
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class  in  the  PVP:  Post-admit  AE,  Pre-admit  event,  and  No  AE  columns  should 
sum  to  100. 


Table  1.  ICD-9-CM  code-identified  adverse  events  confirmed  by  medical  record 
abstraction 


Adverse  event  class  number  and 
label 

Codes 

PVP: 
post- 
admit  AE 

PVP: 

surgery 

AE 

Pre-admit 

event 

No  AE 

N 

% 

% 

% 

% 

1 

Reopen  surgical  site 

43 

49 

42 

19 

33 

2 

Control  postprocedure  hemorrhage 

26 

54 

54 

15 

31 

3 

Perforation  or  laceration 

27 

56 

52 

15 

30 

4 

Septicemia,  bacteremia 

41 

22 

05 

15 

63 

5 

Pneumonia 

52 

50 

42 

13 

37 

6 

Other  infections 

59 

32 

19 

20 

47 

7 

Acute  myocardial  infarction 

43 

47 

37 

14 

40 

8 

Pulmonary  embolism  &  infection 

27 

48 

44 

04 

48 

9 

Heart  disease 

24 

50 

38 

04 

46 

10 

Diseases  of  veins  &  lymphatics,  circ. 
system 

42 

45 

38 

12 

43 

11 

Diseases  of  respiratory  system 

61 

52 

39 

08 

39 

12 

Acute  Gl  ulcer,  bleed,  other  Gl 
disorders 

40 

23 

15 

15 

63 

13 

Postoperative  Gl  disorders 

27 

37 

37 

19 

44 

14 

Nausea,  vomiting,  diarrhea 

59 

53 

46 

07 

41 

15 

Disorders  of  urinary  system 

38 

32 

18 

08 

61 

16 

Complications  peculiar  to  specified 
procedures 

44 

34 

30 

14 

52 

17 

Complications  affecting  specified 
body  systems 

59 

64 

61 

03 

32 

18 

Other  complications  of  procedures 

64 

72 

63 

09 

19 

19 

Complications  of  medical  care,  not 
elsewhere  classified 

26 

58 

27 

08 

35 

20 

Accidental  cut,  puncture,  perforation 
or  hemorrhage 

68 

78 

69 

01 

21 

21 

Other  misadventure  of  surgical  and 
medical  care 

28 

61 

46 

00 

39 

22 

Surgery  as  cause  of  abnormal 
reaction  or  later  complication,  w/o 
mention  of  misadventure 

56 

55 

52 

07 

38 

23 

Other  procedures  as  cause  of 
abnormal  reaction  or  later 
complication,  w/o  mention  of 
misadventure 

58 

48 

38 

09 

43 

Total  codes  and  average  percent 

1,012 

49 

40 

10 

41 

Total  records 

936 
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Results 

The  377  surgery-related  codes  are  shown  in  Table  2  according  to  their  classes. 
They  are  comprised  of  30  procedure  codes,  78  E-codes,  and  269  diagnosis  codes. 
ICD  codes  relating  to  complications  and  misadventures  comprised  the  largest 
group  of  codes.  Codes  in  the  range  of  996.00-999.99  accounted  for  26  percent  of 
the  codes,  followed  by  codes  in  the  range  of  E871.0-E879.9,  which  accounted  for 
21  percent.  Codes  for  digestive  system  diseases  (520.00-579.99)  were  the  next 
most  frequent,  accounting  for  14  percent. 


Table  2.  The  23  AE  classes  and  their  ICD-9-CM  codes 


1 

Reopening  of  surgical  site 

01.23,  03.02,  06.02,  34.03,  35.95,  39.49,  54.12,  54.61 

2 

Control  of  postprocedure  hemorrhage 

28.7,  39.41,  39.98,  49.95,  57.93,  60.94 

3 

Perforation  or  laceration 

29.51,  31.61, 33.41, 33.43,  42.82,  44.61,  46.71, 46.75,  48.71 , 50.61 , 51 .91 , 55.81 , 56.82, 
57.81,  58.41, 69.41, 530.4,  569.83,  575.4,  576.3 

4 

Septicemia,  bacteremia 

038.0,  038.10,  038.11, 038.19,  038.3,  038.40  -  038.9,  790.7 

5 

Pneumonia 

481, 482.0,  482.1, 482.2,  482.30,  482.31 , 482.32,  482.39,  482.40,  482.41 , 482.49,  482.81 
-  482.84,  482.89,  482.9,  483.8,  485,  486 

6 

Other  infections 

008.45,  320.3,  320.82,  321.3,  421.0,  421.1, 421.9,  424.90,  424.91 , 424.99,  510.0,  510.9, 
513.0,  513.1, 519.01, 536.41,  569.61 , 590.10,  590.11,  590.80,  590.9,  595.0,  595.9,  599.0, 
670.00,  670.02,  670.04,  682.3,  682.4,  683,  958.3 

7 

Acute  myocardial  infarction 

410.00,  410.01, 410.10,  410.11, 410.20,  410.21, 410.30,  410.31 , 410.40,  410.41, 410.50, 
410.51, 410.60,  410.61, 410.70,  410.71, 410.80,  410.81, 410.90,  410.91 

8 

Pulmonary  embolism  &  infarction 

415.11, 415.19 

9 

Heart  disease 

423.0,  427.5,  429.4 

10 

Diseases  of  veins  &  lymphatics,  other  diseases  of  circulatory  system 

451.11  -451.9,  453.8,  458.2 

11 

Diseases  of  respiratory  system 

495.7,  507.0,  512.1, 514,  518.0,  518.4,  518.5,  518.81, 518.82,  519.02,  519.09,  519.1, 

519.2 

12 

Acute  Gl  ulcer,  Gl  bleed,  other  Gl  disorders 

530.82,  531 .00  -  531 .21 , 532.00  -  532.21 , 533.00  -  533.21 , 534.00  -  534.21 ,  535.01 , 

535.1 1 ,  535.21 , 535.41 , 535.51 , 535.61 , 536.2,  536.3,  578.0  -  578.9 
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Table  2.  The  23  AE  classes  and  their  ICD-9-CM  codes,  cont. 


13 

Postoperative  Gl  disorders 

536.40,  536.42,  536.49,  564.2,  564.3,  564.4,  569.60,  569.62,  569.69 

14 

Nausea,  vomiting,  diarrhea 

787.01  -  787.03,  787.91 

15 

Disorders  of  urinary  system 

584.5  -  584.9,  598.2 

16 

Complications  peculiar  to  specified  procedures 

996.00  -  996.99 

17 

Complications  affecting  specified  body  systems 

997.00  -  997.02,  997.09,  997.1  -  997.5,  997.60  -  997.62,  997.69 

18 

Other  complications  of  procedures 

998.0,  998.11  -  998.13,  998.3,  998.51,  998.59,  998.6,  998.81  -  998.83,  998.89,  998.9 

19 

Complications  of  medical  care,  not  elsewhere  classified 

999.0  -  999.9 

20 

Accidental  cut,  puncture,  perforation  or  hemorrhage 

998.2,  E870.0  -  E870.9 

21 

Other  misadventure  of  surgical  and  medical  care 

998.4,  998.7,  E871 .0  -  E876.9 

22 

Surgery  operation/procedure  as  cause  of  abnormal  reaction  or  later  complications  without 
mention  of  misadventures 

E878.0-  E878.9 

23 

Other  procedures  as  cause  of  abnormal  reaction  or  later  complications  without  mention  of 
misadventures 

E879.0  -  E879.9 

The  codes  that  comprise  the  23  classes  differ  substantially  from  AHRQ’s 
Patient  Safety  Indicators  (PSIs).  Only  50  of  the  ICD  codes  used  in  the  PSIs  appear 
in  the  Utah/Missouri  classification.  Two  of  the  PSIs,  Postoperative  Sepsis  and 
Accidental  Puncture  and  Laceration,  are  defined  by  the  same  ICD  codes  that 
make  up  Class  4  (Septicemia,  Bacteremia)  and  Class  20  (Accidental  Cut, 
Puncture,  Perforation  or  Hemorrhage)  in  our  system,  although  we  did  not  restrict 
our  sepsis  definition  to  only  those  surgical  discharges  that  were  elective.  Other 
codes  common  to  the  two  systems  do  not  define  PSIs  that  are  congruent  with  our 
classes.  Rather,  codes  defining  a  PSI  tend  to  be  divided  among  two  of  our  AE 
classes,  or  an  AE  class  may  contain  most  of  the  codes  from  a  given  PSI  as  well  as 
codes  not  defining  any  PSI.  Eight  of  the  20  PSIs  are  defined  by  ICD  codes  that 
are  not  present  in  any  of  our  classes. 

As  Table  1  indicates,  the  23  classes  of  codes  had  an  average  postadmission 
PVP  rate  of  49  percent.  The  average  PVP  for  codes  confirmed  to  be  surgery- 
related  was  slightly  lower  at  40  percent.  Codes  failed  to  be  confirmed  largely 
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because  they  did  not  meet  the  hann  or  causation  criteria  and  thus  were  judged  not 
to  be  AEs.  On  average,  41  percent  of  the  codes  in  the  AE  classes  failed  in  this 
manner.  Only  10  percent  on  average  failed  because  the  codes  in  the  class  were 
judged  to  refer  to  events  that  occurred  prior  to  the  current  admission. 

The  best  perfonning  classes  contained  codes  for  complications  (ICD  codes 
996-999).  Class  20,  Accidental  Cut,  Puncture,  Perforation  or  Hemorrhage,  had  a 
postadmit  PVP  of  78  percent  and  a  surgery  PVP  of  69  percent.  (See  Appendix  1 
for  the  results  for  the  individual  codes  that  comprise  each  class.)  As  noted  earlier, 
this  class  corresponds  to  the  Accidental  Puncture  and  Laceration  PSI.  Class  18, 
Other  Complications  of  Procedures,  had  a  postadmission  PVP  of  72  percent  and  a 
surgery  PVP  of  63  percent.  The  best  performing  class  outside  of  those  explicitly 
labeled  as  complications  was  Class  3,  Perforation  or  Laceration;  it  had  a 
postadmission  PVP  of  56  percent  and  a  surgery  PVP  of  52  percent.  Overall,  the 
lowest  PVP  was  for  Class  4,  Septicemia,  Bacteremia,  which  had  a  postadmission 
PVP  of  22  percent  and  a  surgery  PVP  of  only  5  percent.  This  class  is  defined  by 
the  same  codes  as  the  Postoperative  Sepsis  PSI,  as  mentioned  earlier. 

On  the  whole,  the  classes  performed  quite  well  with  regard  to  identifying 
patient  hann  or  injury  originating  after  admission.  Only  1  percent  of  codes  in  the 
best  performing  class  (Class  20,  Accidental  Cut,  Puncture,  Perforation  or 
Hemorrhage)  flagged  patient  harm  present  at  admission.  None  of  the  codes  in 
Class  21,  Other  Misadventures  of  Surgical/Medical  Care,  flagged  patient  hann 
present  at  admission.  The  worst  performers  in  this  regard  were  Class  6,  Other 
Infections;  Class  13,  Postoperative  GI  Disorders;  and  Class  1,  Reopening  of 
Surgical  Site.  Roughly  one  in  five  codes  in  these  classes  referred  to  patient  harm 
present  at  admission. 

Classes  more  often  had  low  PVPs  because  their  codes  failed  to  be  associated 
with  confirmed  AEs;  these  codes  more  likely  reflected  the  patient’s  underlying 
condition  or  disease.  Nearly  two-thirds  of  the  codes  in  Classes  4  (Septicemia, 
Bacteremia),  12  (Acute  GI  Ulcer,  GI  Bleeding),  and  15  (Disorders  of  Urinary 
System)  failed  to  be  confirmed  as  referring  to  AEs. 

PVP  estimates  for  a  large  number  of  individual  codes  were  unreliable  because 
of  their  infrequency  in  the  sample.  For  36  codes,  however,  at  least  10  charts  had 
been  reviewed  to  assess  their  PVPs.  Seven  of  these  had  surgery  PVPs  of  67 
percent  or  better.  These  were  codes  34.03  (reopening  of  recent  thoracotomy  site), 
458.2  (iatrogenic  hypotension),  997.3  (respiratory  complications),  998.11 
(hemorrhage  complicating  a  procedure),  998.12  (hematoma  complicating  a 
procedure),  998.2  (accidental  puncture  or  laceration  during  a  procedure),  and 
E870.0  (accidental  cut,  puncture,  perforation  or  hemorrhage  during  surgical 
operation).  Many  of  the  remaining  codes  appear  promising  but  are  based  on  too 
few  records  to  make  a  judgment.  These  will  be  evaluated  further  in  a  second 
study. 
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Discussion 

Comparison  with  other  studies 

Three  studies  have  utilized  the  CSP  to  investigate  the  ability  of  groups  of  ICD 
codes  to  flag  postadmission  adverse  events  and  substandard  care  in  a  large 
Medicare  sample.10’  n’ 14  The  CSP  uses  trigger  codes  to  assign  patients  to  risk 
pools  (major  surgery,  minor  surgery,  etc.)  and  to  one  of  28  complications 
(reopening  of  surgical  site,  postoperative  acute  myocardial  infarction,  etc.).  A 
computer  algorithm  examines  the  diagnoses,  procedures,  and  procedure  dates  to 
determine  whether  the  record  meets  the  criteria  for  a  preventable,  postadmission 
complication  due  to  substandard  care.  Lawthers  et  al.10  studied  the  extent  to  which 
the  CSP  trigger  codes  were  an  accurate  representation  of  the  charts  and  whether 
they  could  identify  patients  with  AEs  that  occurred  after  admission.  Across  17 
surgical  complications,  they  found  73  percent  confirmed  as  postadmission  AEs. 
The  rate  for  six  medical  complications  was  32  percent.  In  a  second  study  in  this 
series,  Weingart  et  al. 14  examined  how  well  the  CSP  identified  complications  of 
care  and  potential  quality-of-care  problems.  They  found  a  PVP  rate  of  68  percent 
for  15  surgical  complications  and  27  percent  for  5  medical  complications.  Thirty- 
six  percent  of  the  flagged  patients  were  found  to  have  quality-of-care  problems.  In 
the  third  study,  McCarthy  et  al.11  looked  at  the  extent  to  which  complications 
identified  in  the  CSP  were  supported  by  objective  clinical  evidence  in  the  charts. 
For  the  1 1  surgical  complications  studied,  they  found  objective  evidence  in  69 
percent  of  the  charts.  An  additional  12  percent  had  only  a  physician’s  note,  and  19 
percent  had  neither.  The  medical  complications  had  much  lower  confirmation 
rates. 

Of  these  three  studies,  ours  is  most  similar  to  the  Weingart  study.  We  looked 
at  the  ability  of  classes  of  ICD  codes  to  flag  patients  who  had  AEs  that  were  due 
to  care  management  of  the  surgery  patient  during  the  current  admission.  We  did 
not  attempt  to  determine  whether  the  care  was  substandard,  as  was  done  in  the 
Weingart  study. 14  For  15  surgical  complications,  Weingart  et  al.  found  a 
postadmission  rate  PVP  of  68  percent  and  a  substandard-care  rate  of  36  percent. 
Our  postadmission  PVP  for  surgery  AEs  due  to  care  management  across  23 
classes  was  37  percent.  However,  our  sample  was  not  restricted  to  Medicare 
patients,  a  population  that  is  likely  more  at  risk  for  adverse  events.  Also,  our  study 
was  exploratory  in  nature  and  we  did  not  expect  all  the  classes  and  codes  to  have 
high  PVPs. 

Lessons  learned 

Though  an  extensive  number  of  codes  were  studied,  only  a  few  classes  and 
individual  ICD  codes  perfonned  well  enough  to  be  used  to  flag  surgery  AEs  in  a 
statewide  surveillance  system.  Adding  diagnosis  dates  or  readmission  codes  to 
hospital  discharge  records  would  only  modestly  improve  the  usefulness  of  this 
group  of  ICD  codes  for  surveillance  of  AEs,  but  would  help  to  eliminate  false 
positives.  Greater  improvement  for  the  codes  studied  here  would  result  from 
adding  a  marker  to  indicate  that  a  certain  diagnosis  was  the  result  of  care 


254 


Surveillance  Using  ICD-9-CM  Surgery  Codes 


management;  however,  there  is  no  reason  to  believe  such  markers  would  be 
reported  more  reliably  than  are  the  E-codes  for  misadventures.  Further  study  of 
which  codes  best  identify  which  types  of  AEs  in  which  populations,  as  is  done 
with  the  PSIs,  would  be  useful  as  we  continue  to  determine  how  best  to  develop 
statewide  surveillance  of  adverse  events. 

Limitations 

Our  results  were  not  weighted  to  reflect  the  sampling  scheme  due  to  its 
complexity.  Weighting  might  change  the  PVP  percents  somewhat,  but  probably 
not  the  relative  performance  of  the  classes.  We  also  did  not  compute  an  interrater 
reliability  coefficient.  These  have  typically  been  modest  in  studies  of  this  kind.21  23 
The  reviewers  in  this  study  were  not  physicians,  but  rather  nurses  who  were 
carefully  trained  and  tested  prior  to  the  study.  While  neither  nurses  nor  physicians 
typically  have  expertise  in  clinical  coding,  it  is  possible  that  physician  reviewers 
would  have  produced  somewhat  different  results.  A  random  sample  of  hospitals 
would  have  allowed  firmer  generalizations  to  the  population  than  does  our 
convenience  sample,  which  slightly  favored  large,  urban  hospitals.  Smaller 
hospitals  do  not  have  many  surgical  patients,  however,  so  our  results  should  not  be 
overly  unrepresentative  in  that  respect. 

Future  directions 

In  order  to  build  a  more  reliable  ICD-based  surveillance  system  for 
identifying  adverse  events,  the  second  study  in  this  series  will  focus  on  gaining  a 
better  understanding  of  which  codes  to  include  or  exclude  from  the  AE  classes. 

Our  study  found  inherent  weaknesses  in  the  validity  of  some  of  the  defined  AE 
classes,  but  also  stand-alone  strengths  in  particular  ICD  codes.  Inspection  of  the 
classes  with  high  PVPs  found  that  the  scores  were  driven  by  high  PVPs  of  only 
one  or  two  ICD  codes  in  each  class.  Alternatively,  some  low-PVP  classes 
appeared  to  contain  one  or  two  promising  codes  whose  importance  for 
surveillance  purposes  may  have  been  masked  by  their  low  frequency  in  the 
sample. 

Based  on  the  results  of  this  study,  the  followup  study  will  focus  more  on  the 
performance  of  individual  codes  within  the  AE  classes.  This  strategy  will 
facilitate  the  evaluation  of  the  classes  and  the  individual  ICD  codes  for  surveilling 
patient  injury.  Three  classes  (Heart  Disease;  Acute  GI  Ulcer,  GI  Bleeding,  other 
GI  Disorders;  Disorders  of  Urinary  System)  have  been  dropped  completely  from 
the  followup  study.  These  classes  did  have  a  few  codes  specific  to  surgical 
complications,  but  the  codes  occurred  very  infrequently  in  the  discharge 
population  and  would  not  be  very  useful  for  surveillance.  These  three  classes  also 
were  among  the  more  poorly  performing  classes,  with  PVPs  less  than  35  percent 
(Table  1). 

Several  of  the  ICD  codes  occurred  with  low  frequency  in  our  sample  but  are 
fairly  frequent  in  the  discharge  population.  Our  followup  study  will  attempt  to 
obtain  more  reliable  estimates  of  the  validity  of  these  codes  by  oversampling 
them.  Conversely,  those  flag  codes  with  adequate  representation  in  this  study  will 
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not  be  included  in  the  followup  study.  This  method  should  help  us  to  identify  the 
best  classes  and  codes  and  to  assess  the  overall  value  of  the  classes  for  statewide 
surveillance  of  surgery  AEs. 


Conclusions 

The  23  classes  of  ICD  codes  did  only  a  fair  job  of  flagging  AEs  related  to 
surgery.  While  the  codes  tended  to  screen  out  harm  events  that  originated  prior 
to  the  hospitalization,  they  were  less  successful  at  identifying  hann  events  that 
met  our  criteria  for  AEs  related  to  care  management.  Two  classes  and  a  number  of 
individual  codes  performed  well  enough  to  be  used  for  statewide  surveillance  of 
surgery  AEs.  Some  improvement  in  identifying  surgery  AEs  may  be  gained  by 
using  the  flag  codes  with  high-risk  populations,  as  is  done  with  the  PSIs.  Further 
study  is  needed  to  refine  the  classes  and  to  evaluate  their  use  for  statewide 
surveillance  of  surgery  AEs. 
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